[Feature] End test runs faster - #130

sequenceDiagram
  participant RunCommand
  participant TestRunSocket
  participant sync_log_file_from_backend
  participant AsyncApis
  participant write_log_file_from_entries
  RunCommand->>TestRunSocket: Await websocket task
  TestRunSocket-->>RunCommand: Complete after run finishes
  RunCommand->>sync_log_file_from_backend: Synchronize local log
  sync_log_file_from_backend->>AsyncApis: Poll state and download persisted log
  AsyncApis-->>sync_log_file_from_backend: Return state and log entries
  sync_log_file_from_backend->>write_log_file_from_entries: Write validated entries
  write_log_file_from_entries-->>sync_log_file_from_backend: Replace local log

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to 386fe

Some valid backend logs may fail to replace an incomplete local log, and the timezone tests may affect later tests. These bounded issues should be fixed or explicitly accepted before merging.

• Fix all pre-merge checks with AI

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

If the backend goes down during the test and the db write doesn't happen, the CLI-generated log file does not get replaced.

Hi @greens Nice improvement. A possible follow-up, not for this PR: since the CLI has its own endpoint (POST /test_run_executions/cli), the backend could skip broadcasting log records over the websocket for CLI-created runs and only store them in the DB. The CLI would keep the websocket for state updates and fetch the full log at the end.

Does that make sense to you, or am I missing something about how the CLI uses those log records? Happy to hear your view.

Is this related to this issue project-chip/certification-tool#1157 ?

@rquidute The primary issue is the one I mentioned in my comment: if the backend crashes during the test, you would get no logs, because the logs written to the DB are put there all at once after the test completes. It would also mean that the live log viewer on port 8998 is gone.

One thing your question made me consider is whether there are any logs from the CLI itself that are not in the DB, and it seems like there are a couple, primarily around camera/ffmpeg handling.

Hi @greens, thanks for the explanation. My idea was to use the DB log as the only source and keep the websocket for state updates. The crash case rules that out, because the DB log is only written after the test completes. The live log viewer on 8998 is also fed by the websocket log records, so it would show nothing during the run. I'll leave it.

One question about the CLI-only lines you mentioned. The camera handlers and ffmpeg_converter log through the same loguru sink, so they currently end up in the local file, and the DB-log replacement would drop them, as far as I can tell from the code. They're not test output, but things like "Stream readiness timeout" or ffmpeg errors can help explain why a camera test failed. Do you think they're worth keeping in the file, or is the DB log enough?

@rquidute Yeah. I think they're worth keeping, so I'm going back to the drawing board a bit with these changes.

@rquidute I think, with the additional work to include the CLI-only log files, it made more sense to go with this much simpler change. I'll have a corresponding backend commit, but now tests will only report that they're finished (and hence the socket will close) when the logs have finished flushing. Older backends will still get the drain timer.

This splits the changes across two repos, but I think it ends up being a simpler solution. It should also still allow for your improvements in project-chip/certification-tool-backend#389